Skip to content

Feat: generate llms.txt and markdown pages for AI agents - #1470

Merged
anoop2811 merged 2 commits into
kubevela:mainfrom
anoop2811:feat/llms-txt
Nov 12, 2026
Merged

anoop2811 merged 2 commits into
kubevela:mainfrom
anoop2811:feat/llms-txt

Conversation

@anoop2811

@anoop2811 anoop2811 commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

Description of your changes

Makes the documentation consumable by AI agents: generates an llms.txt index and a markdown copy of every docs page at build time.

  • docusaurus-plugin-llms (new dev-time plugin) emits llms.txt, llms-full.txt, and a .md twin next to every page's HTML, so agents can fetch <page-url>.md instead of parsing themed HTML. Version output mirrors the site's routing: the root llms.txt indexes the latest release served at /docs/, while next and older versions get their own indexes under /docs/<version>/llms.txt.
  • scripts/llms-postprocess.js (runs via the postbuild hook, so CI and Netlify-style yarn build both trigger it) restructures the indexes into the llmstxt nested layout: CLI man-pages move to /docs/cli/llms.txt and other versions hang off /docs/versions/llms.txt. This keeps every index in the 77–91K character range — under the ~100K budget agents can ingest in one read — and the script fails the build if an index ever crosses it. It also prepends an index pointer to every generated markdown file.
  • src/theme/Root.js adds a visually hidden, server-rendered note on each page telling agents where the index and markdown variants live.

Measured with afdocs (the scanner behind CLOMonitor's agent readiness section) against a local production build: overall 94/100, with content discoverability at 92 and index coverage at 100%. The one category that still fails — Accept: text/markdown content negotiation — is a hosting limitation, not something the repo can fix.

I have:

  • Read and followed KubeVela's contribution process.
  • Update sidebar.js if adding a new page.
  • Run yarn start to ensure the changes has taken effect.

Special notes for your reviewer

No docs pages or sidebar entries change — all new output is generated into build/ (verified with full yarn build runs rather than yarn start, since the plugin and postbuild hook only run on production builds). Build time grows by roughly 1–2 minutes for the markdown generation across versions. The commit is DCO signed off.


Summary by cubic

Makes the KubeVela docs consumable by AI agents: builds an llms.txt index and a markdown copy of every page at docs build time so agents can fetch https://kubevela.io/llms.txt or any page's .md twin instead of parsing styled HTML. Measured 94/100 on the CLOMonitor agent readiness scan (up from no index at all), with content discoverability 92 and index coverage 100%.

Build changes

  • Adds the docusaurus-plugin-llms dependency and configures it to emit per-version indexes matching the site's routing (/docs/ for the latest release, /docs/<version>/ for next and older versions).
  • Adds scripts/llms-postprocess.js as a postbuild hook that restructures the plugin output into the llmstxt nested layout: CLI man-pages move under <version>/cli/llms.txt, per-version indexes under /docs/versions/llms.txt, and every index stays under the ~100K character agent ingestion budget (the script fails the build if one exceeds it).
  • Chains the postprocessor into the standalone Dockerfile build, which runs docusaurus build directly and would otherwise skip yarn's postbuild hook.
  • Prepends a pointer to the root llms.txt on every generated markdown page and adds a visually hidden note to every HTML page via src/theme/Root.js.

No docs pages or sidebar entries change; all output is generated into build/. Build time grows by roughly 1–2 minutes for markdown generation.

Written for commit 2c722c3. Summary will update on new commits.

View guided diff

- Add docusaurus-plugin-llms to emit llms.txt indexes and a markdown
  copy of every docs page next to its HTML (<page-url>.md)
- Mirror the site's version routing: the root index covers the latest
  release, with nested indexes for the CLI reference, next and older
  versions under /docs/<version>/llms.txt
- Keep every index under the ~100K character budget agents can read in
  one fetch; the postbuild script fails the build if one grows past it
- Point agents at the index from every page: a hidden note in the HTML
  (src/theme/Root.js) and a blockquote atop each generated markdown file

Signed-off-by: Anoop Gopalakrishnan <anoop2811@aol.in>
Copilot AI balanced review requested due to automatic review settings November 12, 2026 19:00

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 5 files

Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic

Comment thread package.json Resolved
- The self-contained Dockerfile invokes docusaurus build directly,
  which skips yarn's postbuild hook; chain the postprocessor so every
  build path produces the same output (CI already uses yarn build)

Signed-off-by: Anoop Gopalakrishnan <anoop2811@aol.in>
@anoop2811
anoop2811 merged commit 0a18e2f into kubevela:main Nov 12, 2026
7 checks passed
@anoop2811
anoop2811 deleted the feat/llms-txt branch November 12, 2026 21:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants